SABES Philiip UCSF Multisensory Integration 11 Mar 2013

pronounced  SAY bus... (note: he goes by SKIP (not Phil))

dept of physio and center for integrative neurosci
Sloan-Swartz  Center  UCSF

opens with   video

monkey controlling a robot arm  to grab a target

also several preclin studies in humans...

hochberg/ donoghue 2012 lab

video...   tetraplegic woman grabbing a drink bottle with robot

Neuroprosthetics...

but something very unnatural with the quality of the movement
...  so, what's the problem?

Is it that....
not recording from right neurons, not decoding right..?

but also  subjects must rely on visual feedback (main point)

slow, delib  mvmts  reminiscent of deafferentation...

successful BMI will require sensory fdbk...

2010  even passive proprioceptive feedback helps..

see Nico Hatsopoulis  2010 study...

***

he says  give feedback using  central stimulation  (to provide feedback during BMI)

but  cannot invert the biophysics  ...
topography is not clear... (how and where to stim)

our approach:  take advantage of brain's natl platicity

each time u make a 20% error , correct it on the next mvmt

error corrective learning  cheang and sabes  2005/06

bayesian integration of past experience  verstynen and sabes 2011

what should u do when vision and propriocep DISAGREE?

first, sensory integration...

vis guided mvmt...

u can see where limb is and feel it also

want a combined estimate of where u are...
the combo knows abt uncertainty...

he shows a spectrum of info from vision to proprio with a combo estimate  in btwn

do a variance weighted estimate...

having two sensors  drives a continual shift in reliability as errors change...
yet the system adapts:
sensors are always noisy...

merge  the signals toward one another:
he uses a variance weighted estimate.

real sensory signals are  complex and heterogen

so, how does brian do this???

must be learned....  how does it work...?

he will tell a story abt how he thinks it works...  just a guess, a model...

Sensory integration as statist model-building...

how does brain know how to extract out relev measure
 from visual space and from propriocep space...

some nonlinear mapping btwn proprio and visual space...

take a  ML = machine learning approach

build a generative model...
(he shows such a model that includes...)

gain factors  which influence firing rate of neurons that repres  S  =  spatial locations...

from Geoff Hinton...

 RBM  restr  boltzmann  machine

uses contrastive divergence  (frim Hinton)
a Hebbian learning rule  +  antihebbian rule

use poisson units...

analytically solve for where the limb was

feedback to input to get a multsensory decoder  (just center of mass)

Network integrates optimally

MAP  = maximum a posteriori estimate  
(what's the best u can do given both proprio and vision)

RBM  = network based estimate

result... it is almost as good as u can do...
learning to do the optimal thing....

if we had 2N units, it would be perfect (identity matrix)

N units is about the right number  (is  hidden layer  has same # of units as the sensory layer which has 2N
(one N for proprio and one N for vision)

network also keeps info abt uncertainty... the variance...  per trial..

vision =  joint space in  theta and radians (angle)

vs  shoulder angle ,  radians

in cortex info is combined in hierarach fashion

can  take output of first  network and integrate that with still another brand new network...
cuz it keeps uncertainty...

also know where eyes are looking...  

Conclusions so far:

a network that learns de novo the statist relations
  can also do multisen integration
bayesian estim of priors
hierarch combo of info
and sensory translation

see Yildrn and Jacobs  from Vis Res  2012...

areas around IPS  might be the ones that combine this info  to so sensory integration

combined  vision  V2,MT  from vision that goes to PO(V6)  then to VIP

with info  from  area 3  proprio etc.

does the RBM look like parietal cortex?

comparing model to physio... it's sort of right...

see some neurons that repr  body and other that repr eye...

Richard Anderson....

important idea:  the job of the multisens cortex is to extract statist regularities from its inputs

eg if two modalities correlate

learn to combine signals efficiently...

****

artificial sensory feedback...

are statist correl enuf to drive sens integration...

neural prosthetics...

in reality, all prosthetic devices require learning  (even cochlear implant)

we want to focus on learning...

to make the point, use an arbitrary input...

use ICMS  feedback = intracortical microstim

use random dots kinetogram...

change reliability by coherence  a la Bill Newsome...

have monkey reach an unseen tareget

delay reach task...

pair vision with ICMS...

ultimately... can monkey do it with just ICMS

(its like colder / hotter game as a kid)

cortical stim...

take utah rate...  stimulate  the somatsens cortex...

when monkey is far  do  fast stim...
when closer... slow it down  (hot)

cosine-tuned trodes...
if going toward E1  the rate incr there...

monkey can do this task with ICMS alone....

simply by pairing  ICMS and vision u can learn ICMS  alone...

what happens with both cues...

u can use BOTH and  performance is even better  (if  visual coherence is poor)

people and we (in our model)  do  minimum variance integration

variance of combined is smaller than with either alone...

summary:  learning driven by correlated inputs  (vision and ICMS)

get  sensory substit of vision by ICMS
and sens augmentation of vision by ICMS..

we showed that the  optimal  (minimum variance) estimate is 
 the natural way that 2 modalities are combined...

want sensory-motor prosthetics
to get greater sense of embodiment...

we expect a leap in performance...

DARPA Repair project..

jay:  Q about meshing differences in timing:
A :  proprio feedback is much faster than visiual
visual take > 100 ms
proprio  2X as fast...
that's the easy answer...

v strong monosynap projections from S1 to M1...

proprio sensory  are in same  coordinate  frame as  movements...

representation are NOT simple in neurons... these combos MUST be learned...

representations are not simply additive...

LIP  for saccades  etc.

Q  what is RBM  doing?

just trying to build a repres of statistics...

Q  where does the implant electrode need to be implanted.. does it need to be in sens/motor cortex

A  do not have great data...  but  he guesses  early sensory area...

BMI  something special abt  somatosens cortex...  want to go to S1... special role in
sensory motor coord...